Method of replicating data between computing devices which each use local clocks

ABSTRACT

Where a first computing device is given responsibility for determining whether data that is time stamped by a second computing device is replicated or not, then the first device can compare a time stamp from the second device against a time signal from its own internal clock to determine a delta and use that delta to deduce the correct delta to apply to time stamps associated with later data from the second computing device.

FIELD OF THE INVENTION

This invention relates to a method of replicating data between computingdevices which each use local clocks.

DESCRIPTION OF THE PRIOR ART

Data replication between computing devices occurs in many differentcontexts, such as synchronising documents, e-mails, contacts andcalendar entries across different computing devices used by the sameperson. In order for data replication to be accurate, there must be somesystem for deciding which of two or more inconsistent data replicationrequests should be acted on. Take for example the situation where, attime 1, a user enters a change to his e-mail details for person X on hismobile telephone. Some time later, at time 2, the user realises thatthose changes were in fact wrong. He makes a further change to thoseperson X email detail, but does so on a PC. At time 3, he synchroniseshis PC to a master server, updating the master record of person X'se-mail to the correct address. But suppose later still, at time 4, hesynchronises his mobile telephone with the master server—this wouldpotentially over-write the correct person X email details (entered attime 2) with the incorrect details entered originally at time 1.

To address this problem, synchronisation systems typically employ timebased arbitration rules, such as giving priority to the latest recordedchange (time 2 as compared to time 1 in the example above). Thisrequires a device to be able to associate a time stamp with data toindicate when that data was changed on the device; this is commonly donefor wire-line based devices (as opposed to devices communicating over awireless network) by providing a database on a computing device whichstores data of a given type (e.g. a database for all contacts detailsfor an individual—name, e-mail, telephone, mobile etc; these form asingle contact record. There would typically be another database fore-mail, another for calendar entries etc.). Whenever anything in a givenrecord is changed (e.g. a new e-mail address for an individual), thenthe device logs that change as part of the record and time stamps it(again, as an entry forming part of the record) with the timeestablished by a quartz clock running locally on the device. Changes todata (e.g. the new e-mail contact details for the individual) are thensent to a central server; an entire replacement record is usually sent(e.g. all existing contact details for the individual, plus the newe-mail address, plus the time stamp)—the central server then updates itsmaster copy of the information and sends updating information to anydevices so that all maintain the same information. The central servertypically applies arbitration rules using the time stamp to resolveconflicts between competing and inconsistent synchronisation requests.

Conventional data replication systems are therefore reliant on timestamps applied to database records. The time stamps are made by a localclock running on the device into which the changed data was first inputThis device could be a client device, server or peer. When there areseveral devices (each with a local clock) which can accept changes tothe same data, then a single device is nominated to apply time basedarbitration rules using the time stamped database records. A problem isthat those local clocks applying the time stamps will rarely besynchronised to each other. This can happen because (a) quartz clocktimes drift, (b) users can deliberately input small time adjustments(e.g. so that the device clock runs 5 minutes fast) and (c) users caninput deliberate time adjustment of±n hours (e.g. moving timezone/daylight saving). This can lead to competing and inconsistentsynchronisation requests that cannot be correctly handled by the normalarbitration rules.

One approach to solving this is to require that all devices regularlyconnect to an external and accurate clock—e.g. a web based clock givingGMT (or applicable local time) and re-set their clocks using this. Manyusers are however reluctant to do this; further, there is no guaranteethat all devices providing, to a central arbitrator, data to bereplicated will in fact be mutually time synchronised.

SUMMARY OF THE PRESENT INVENTION

In accordance with a first aspect of the present invention, there is amethod of replicating data between computing devices which each uselocal clocks, in which a first computing device is given responsibilityfor determining whether data that is time stamped by a second computingdevice, using a clock running on the second device, is replicated ornot;

wherein that first device can compare the time stamp from the seconddevice against a time signal from its own local clock to determine adelta and use that delta to deduce the correct delta to apply to timestamps associated to later data from the second computing device.

In this way, the first computing device uses its knowledge of thecorrect deltas relative to its own internal clock in order to applycorrectly a time dependent arbitration rule to determine which ofseveral inconsistent data replication requests (e.g. from several seconddevices; from the first computing device and a second device etc.) itshould action. Local clock drift can be fully compensated for.

This approach useful when the time stamps applied to the later data bythe second device are not received immediately when generated butsignificantly later, so that the first computing device cannot directlycompare the time stamps against signals from its own clock. This willoften arise in practice, especially where the second computing devicesare portable, wireless devices (e.g. mobile telephones) and they batchup all data to be replicated for efficient use of bandwidth, to uselower cost night time tariffs or are simply out of coverage when thechanges are first input.

In one implementation, the internal clock of the first computing deviceis set to an absolute, externally derived time, such as GMT. Otheroptional features are that the delta deduced by the first device assumesthat the local clock in the second device is drifting in a predictablemanner, such as a linear manner.

Further, if a deliberate time adjustment to accommodate a change of timeframe (e.g. new time zone/daylight saving) is made to the local clock onthe second device, then the second device can store a record of thistime adjustment and provide it to the first computing device. The firstcomputing device can then compensate for deliberate time adjustments andcorrectly relate time stamps sent to it from the second computing deviceto its own internal clock, and hence apply time based arbitration rulesconsistently for all devices connecting to it and supplying time stampeddata replication requests. The second device can also alter one or moretime stamps stored on it by the amount of the deliberate time adjustmentso that the time stamps remain correct in the new time frame.

In a second aspect, there is a computing device capable of replicatingdata and having responsibility for determining whether data that is timestamped by a second computing device is replicated or not;

wherein the device can compare the time specified by a local clock onthe second device against its own local clock to determine a delta anduse that delta to deduce the correct delta to apply to time stamps forlater data from the second computing device.

Section 6.2 of the following Detailed Description describes animplementation in more detail.

DETAILED DESCRIPTION

The present invention will be described with reference to animplementation from Cognima Limited of London, United Kingdom. Cognimahas developed a data replication technology that directly addresses theneed for Mobile Service Providers (MSPs) and Network Operators toincrease consumer adoption of data services, encourage greater loyaltyfrom their valuable customers, and differentiate their services from thecompetition.

Cognima's data replication solution addresses these issues by:

Increasing adoption by making data services compelling and effortless touse.

Establishing a high barrier to churn by securely backing up subscribers'personal data on servers controlled by those subscribers' MSP.

Enabling the MSP to create differentiated services by controlling thecustomer experience.

1. Overview of Uses for the Cognima Data Replication Framework

Cognima's data replication framework enables a Mobile Service Providerto build compelling services for consumer markets. The MSP hosts aCognima Server at its data centre. The server comprises an Oracledatabase plus Cognima's multi-threaded Java communications server,hosted on a standards-based J2EE application server and carrier-gradeUnix hardware. Section 4 and later sections describe the technicalimplementation in detail.

The Cognima framework replicates data entered in a mobile phoneautomatically (without any user intervention) to other phones via theCognima Server. Similarly, data from external systems connected to theCognima Server is automatically kept up-to-date on mobile phones.

Mobile subscribers using Cognima-enabled applications experience analways-available, instant connection to their personal information andfriends.

Personal information can include the subscriber's address book,messages, bank account details, stock prices, pizza orders, calendar,current traffic on a route to work, or any other personalised content.The data is always kept securely backed-up on the Cognima Server andautomatically replicated on all relevant client devices.

Always-available means that the personal information is accessible onwhichever device or handset the subscriber is carrying, whethercurrently connected to the network or not since the user can alwaysaccess personal information stored locally on the device). Users canalso edit and manage their personal data directly on the server via aweb interface—the Virtual Phone.

Instant means that subscribers do not have to wait for data to downloadfrom a server; the latest information is on their handsets even beforethey know they need it since that data is automatically sent to thehandset (e.g. polling by the handset may occur; this can be regularperiodic—such as every 30 minutes or at pre-defined times (4 pm, 5 pmetc). Pushing to the handset may also occur).

Subscribers can share their data across multiple devices and with theirfriends since the Cognima Server can replicate this data to any defineddevice or defined individual.

1.1 Example Cognima Applications Customer Need Cognima Application SarahSarah's phone has been Whenever Sarah enters data in stolen, includingsome her phone, Cognima important contact automatically backs it up on anumbers and messages central server at the MSP's for which she has madedata centre. Sarah can buy a no manual back-up new mobile phone, andcopy. retrieve all her contacts and messages instantly from the centralserver, as long as she remains with the same MSP. She can also deleteher data from the stolen phone via the MSP's portal. Jill Jill is outshopping. Cognima keeps Jill's Before making an personalised contentexpensive purchase, she (including her bank account needs to know if herdetails) up-to-date salary has been paid into automatically on hermobile her bank account. phone by periodically (or at a However, she isin the predefined time or even basement of a immediately a changeoccurs) department store, and sending any changed data to has no networkJill's mobile. The latest data is coverage. there on Jill's phone evenbefore she knows she needs it. She can access it instantly, even ifthere is no network coverage. Matthew Matthew likes to keep Cognimashares Matthew's his friends informed presence profile with his abouthis current friends. When he changes his availability and ‘mood’.profile (e.g. selects an icon to He also likes to see what indicate he'sfeeling sociable) his friends are up to. the icon updates automaticallyHe's mainly interested in in Matthew's address book keeping track ofwhat's entry on his friends' phones. happening in his social Matthew cansee presence group, and he wants to information for all his friends dothis at a glance, at a glance on his own phone. without having to go Hecan even ask his phone to ‘on-line’ or send lots of alert him when afriend is expensive messages. feeling sociable or bored, so that he canimmediately call. Laura Laura has two mobile Cognima automatically keepsphones - one she uses at all the data in Laura's phones work, and afashion- in step. Whenever she edits phone she takes out in data on onehandset, it is the evenings. She wants immediately (or periodically orto keep the same at a predefined time) replicated address book on bothonto the Cognima server devices, but she hates which then updates herother entering data twice, and phone as well. She never has she's neverfigured out to remember to press a ‘sync how to use the sync button’ -it just happens. Jill software that came with even shares some of theher phone. Swapping contacts in her phone with her the SIM card over ishusband, Geoff. When Geoff cumbersome, and leaves enters his mother'snew behind data in the mobile number, it is phone memory. automaticallyupdated in Jill's phones as well. Juha Juha also has two mobile WithCognima, SMS, e-mail devices - a phone and a and other types of messageswireless-enabled PDA. can be read and sent from any He needs to read anddevice, and also using a reply to e-mail and SMS ‘Virtual Phone’ webinterface. messages on both Messages are received on all devices, but hegets devices used by the subscriber, confused and frustrated, and sentmessages appear in and loses productivity, the Outbox on all devices.when his Inbox gets out Any message read on one of sync. device isinstantly marked as read on all other devices. Messages deleted from amobile phone can be stored and retrieved via the Cognima Server.2. Benefits to the Mobile Subscriber

Cognima provides an ideal framework for implementing mass-marketconsumer data services based on the following key benefits:

Friendliness: no user intervention is required. Subscribers never needto press a ‘sync’ or ‘download’ button to access their data. Systemconfiguration and secure data transfer are completely transparent to theend user.

Instant availability: the user is always able to interact instantly withlocal data (even when off-line), whilst any updates take place silentlyin the background. For example, users can read their personalisedcontent whilst on an underground train. The user experience is separatedfrom the data transport.

Affordability: The MSP can control when replication takes place, and theQuality of Service (QoS) delivered. However, because the user experienceis separated from the data transport, lower QoS does not affect theuser's perception of the service. Crucially, this allows the MSP tooffer low-cost, subscription-based services with relatively poor QoSwithout sacrificing user experience—e.g. data replication can happenovernight for non-urgent data services such as bank statements, yetstill be satisfactory to users. Overnight data replication usesotherwise underused bandwidth and is hence far cheaper than peak timedata replication. Urgent data replication (e.g. presence information)can happen at any time on a periodic or (optionally) continuous (push)basis and attract a higher charging rate. Furthermore, efficient use ofphone memory & processor power allows Cognima client software to becost-effectively installed in even the cheapest mass-market phones.

3. Benefits to the Mobile Service Provider

Cognima presents a MSP with a means to generate new data revenues,reduce churn, and to differentiate its services from those of itscompetitors.

3.1 Increased Usage of Existing Mobile Services

Cognima increases usage of existing mobile services:

Messaging and content-based services become much more convenient andimmediate, and will therefore be used more.

The enhanced immediacy of presence information increases the use of chatand Instant Messaging, and an alert when free capability will boostvoice calls.

Effortless management of multiple devices allows users to carry anappropriate phone on any occasion, and therefore make more calls andsend more messages.

3.2 Compelling New Services

Cognima enables rapid introduction of compelling and affordable newmobile data services.

Cognima delivers a compelling user experience for new services inlow-end phones using only spare network capacity. This is affordable andscalable for the network operator, allowing the MSP to offerunderstandable and predictable pricing for mass-market subscribers.

Most of the application development for new Cognima services takes placeon the server side, allowing the MSP to bring new services to marketquickly.

Cognima's client software can be installed as a flash memory upgrade,endowing today's mass-market handsets with smart-phone-likecapabilities. New software applications can be downloaded over the airto existing Cognima-enabled handsets, allowing MSPs to roll out new dataservices without waiting for new devices to support them.

Third party application developers can leverage the MSP's Cognimainfrastructure to develop new applications for the MSP's network.

3.3 Churn Reduction

Cognima services act as a significant barrier to churn. For example, asubscriber who stores their personal information securely at their MSP'sCognima Server can buy a new phone and immediately retrieve all personalinformation to their new device. All this personal information may belost if they decide to take out a subscription with a different serviceprovider.

3.4 Differentiation

Today, subscribers have the same basic experience of using mobile dataservices on all networks. For example, the experience of using WAPservices is defined by the WAP protocols, the browser in the phone, andthe content accessed. Many MSPs have realised that they mustdifferentiate themselves by giving their subscribers a unique userexperience, but are hindered from doing so by severe constraints tocustomising the services in mobile handsets.

Cognima gives MSPs the ability to implement services on the handset, andthereby to regain control of their subscribers' user experience. Mostimportantly, Cognima allows this without sacrificing interoperability;support for industry standards is achieved through straightforwardintegration with the Cognima Server. The net result is that the MSP'sposition in the value chain is strengthened versus the powerful brandsof handset manufacturers and content providers.

4. Cognima Data Replication Framework Functional Design

4.1 Introduction

This and subsequent sections of the Detailed Description are intended todescribe how the Cognima data replication system actually works. Itcovers the behaviour of client devices, the Cognima Server and the webclient, without going into details of specific hardware, programminglanguage, software class design or environment. It does describe thebasic data structures and algorithms used.

Terms Client device A phone, PDA or other machine running the Cognimaclient software. Cognima A server accessible by client devices whichruns the server Cognima server software to replicate data. ReplicationThe process of copying data from a client device up to the CognimaServer and then down to other client devices belonging to the same user.User A human being who owns and uses at least one Cognima client deviceUser data The set of information (contacts, messages, ringtones,pictures etc) that a user might want to store and manipulate on a clientdevice.4.2 Purpose

The objectives of the Cognima software are:

To allow a user instant access to view and modify an ‘up to date’ copyof their data on multiple handheld devices capable of wireless dataconnectivity.

To allow a user to view and modify the same data using a conventionalweb browser.

To effortlessly provide secure backup of a user's data.

To give a user powerful data functionality on a cheap handset bydisplacing complicated and expensive processing to a server.

4.3 Highest Level Description

Client devices hold a copy of the user's data in a database on theclient device. The user can access this data whether or not he has anetwork connection and therefore always has instant access. When a userchanges the data on his device, the changes are copied to a Change-Log.The client device connects periodically to a Cognima Server on thewireless network, to send up the changes from the Change-Log and receivenew data. This separates the act of changing data from the need toconnect to the network (i.e. push is not continuous in a preferredimplementation). The Cognima Server updates its own database with datachanges received from the client device, and populates Change-Logs forany other devices the user owns. When these devices next connect, theywill receive the changes and thus the devices are kept in sync, eachwith a copy of the same data.

The Cognima Server contains a web server which allows the user toexamine directly using a web browser the copy of the data held in theCognima Server database, and make changes to it as he would on a clientdevice. The Cognima Server also acts as a gateway for the user tocommunicate with other servers on the network/internet. For example, theclient device can effectively ask the Cognima Server to send a messageas an SMS or an email or a fax by setting a few flags in a messageobject and the Cognima Server contains the functionality to communicatewith email servers, SMS servers and fax machines. This can be extendedto servers holding ringtones, banking details, games etc. It is easierand cheaper to build the software on the Cognima Server to talk to theseother servers, than it would be to build the software on the clientdevice.

5. Lower Level Concepts

5.1 Data Structures

5.1.1 Ids

Cognima user data is described using the terminology of objectdatabases: classes and objects. Unfortunately, there is room forconfusion with similarly named OO programming concepts and caretherefore needs to be taken.

All users in a Cognima network are assigned a user id. This id is uniqueto the network—i.e. provided by a given network operator. All users havea Cognima address which is a combination of their user id and CognimaServer URL. This is unique in the world. Each device which belongs to auser is assigned a device id. The device id is unique to the user. Thisis only 8 bits so a user can have a maximum of 253 devices (id 254 isreserved for the web, id 255 is spare, id 0 is invalid). All user datais classified into classes (contacts class, messages class, banktransactions class etc) and the classes are assigned a class id which isunique in the world. Class id ‘12’ refers to a contact, for example.

An instance of a class is an object, which is assigned an object idunique to the user, e.g. a contacts class object might be the contactfor “John Smith”. The object id is generated by concatenating the deviceid of the device which created the object with a monotonic increasingcount which increases over the life of the device. So each device cancreate a maximum of 16777215 objects (if we encountered this limit wecould reset the device id). Classes are defined by the properties whichconstitute them. A class is essentially an array of properties. Eachproperty in the class has a property id which is unique to the class(and is actually just the array position of the property in the propertyarray, starting from zero).

5.1.2 Creating Objects

An object is created on a device. It is assigned an object id and savedto the device database. A copy is also saved into a Change-Log. When thedevice next connects to the Cognima Server the entry in the Change-Logis sent up. The Cognima Server saves the object to its database(recording the system time), does any class specific processing that maybe required (such as generating and sending an email) and adds entriesto Change-Logs for any other devices that the user may own which havedeclared interest in the class. (The entries should be for the correctversion of the class on the device).

An object may also be created on the web portal. The object id isgenerated (using device id of 254 as described above) and processedidentically to the device. There is no Change-Log for the web portal, itgets selections directly from the Cognima Server database.

An object may also be created by a server application (e.g. a messagingmodule might receive an email from which it creates a message object).The object id is generated (using device id of 254 as described above)and processed identically to the device.

5.1.3 Updating Objects

One or more properties of an existing object are modified on a device.The changes are saved to the device database. Each changed property isused to generate an entry in the device Change-Log. These are sent up tothe Cognima Server.

If the time of the update is later than the ‘last changed’ time for theproperty in the Cognima Server database then the Cognima Server savesthe changes to its database (recording the new ‘last changed’ time forthe property), does any required class specific processing and addsentries to Change-Logs for other devices which belong to the user, havedeclared the class and have a version of the class which contains theproperty. The update is also placed on the Change-Log for the devicethat originated the change. This may seem strange but is required tocope with the following scenario:

A user has 2 devices A and B. He updates property 7 on A offline at 5 pmand updates it on B offline at 6 pm. He connects to the network with Afirst. The valve of 7 on A gets put in the Change-Log to be sent to B.Later B connects. Its value of 7 is more recent so the value of 7 on Bis sent to A, but B gets A's value. Replicating the value of 7 back to Bfixes this.

If an update is received by the Cognima Server for an object which ismarked as deleted and the update is later than the deletion, then thisis interpreted as an un-deletion. The object is undeleted, updated andthen a refresh of the object in placed on the Change-Logs for allappropriate devices. Updates from the web portal or server applicationswork in the same way.

5.1.4 Deleting Objects

An object is deleted on the device. It is removed from the devicedatabase and an entry is put on the Change-Log listing the class id andobject id. The entry is sent up to the Cognima Server.

If the time of the deletion is later than the last updated time of theobject, then the Cognima Server marks the object as deleted in itsdatabase, does any class specific processing and adds the entry to otherdevices that belong to the user and have declared the class.

If the time of deletion is earlier than the last updated time then thisindicates that the deletion is invalid and a refresh of the object isput on the Change-Log for the device which originated the deletion.

The deleted object is viewable in the web portal a manner that makes itsdeleted status clear. The user can select the object for un-deletion.The deletion mark is removed from the object in the Cognima Serverdatabase and entries to refresh the object are placed on the Change-Logsfor all devices that belong to the user and have declared the class.

5.1.5 Property Types

Each property has a type. There are currently 9 permitted propertytypes: Type Type name value Type description KcogTypeRef 0 4 byte objectid of another object KcogTypeInt 1 signed 4 byte integer valueKcogTypeUInt 2 unsigned 4 byte integer value KcogTypeFloat 3 signed 4byte floating value KcogTypeStr 4 a CogString (a 4 byte unsigned integerholding the number of characters in the string, followed by thecharacter bytes) KcogTypeTime 5 unsigned 4 byte integer value indicatingthe number of seconds since midnight 1st Jan 1990 KcogTypeTypedStr 6unsigned 4 byte integer value followed by a CogString KcogTypeBlob 7 astream of bytes preceded by a 4 byte unsigned integer which holds thenumber of bytes KcogTypeArray 8 a blob structure which can hold an arrayof any kind of data

A CogString is a character count followed by the characters. If thestring is ASCII then the space taken up by the string will be (4+charcount) bytes. If the string is Unicode then the space taken up will be(4+(char count * 2)) bytes.

A CogTypedString is a CogString preceded by a type (4 byte unsignedinteger). The only use of a typed string so far is a Contact Point. Thetype identifies the type of contact point (e.g. email address, homephone) and the string holds the address (e.g. bob@xxx.yyy, 01233556677).

A CogBlob is a length in bytes followed by that number of bytes. It canbe used to store any binary data

A CogArray is passed around as a 4 byte unsigned integer ‘type’ followedby two blobs. The ‘type’ indicates the type of elements held in thearray. The first blob is an index blob: it holds a sequence of offsets(4 byte unsigned integers) into the second blob. The second blob is thedata blob which holds the elements of the array as a sequence of binarylumps. Elements can be extracted from the data blob by counting alongthe index blob to get the offset of the start of the element in the datablob. This is the stream structure of the CogArray as it is passedaround. Inside a particular system it may appear as a conventionalvector (i.e. already parsed).

The only implemented example of a CogArray is the MessageAddress. Eachelement of the MessageAddress is an AddressPair. An AddressPair is acontact id (object id of a contact object) followed by a Contact Point.

5.1.6 Smart Property Parameters

Some of the properties can be made “smart”. This means they can beparameterised for a specific device to sculpt the data in the propertyfor the characteristics of the device. In practice the parameters aretwo 4 byte unsigned integers, one is a smart type and the other is a maxsize. For example, the property which holds the body text of a messagemight be parameterised to smart type kCogPlainText and max size 100 on acheap phone with limited memory, but parameterised to be smart typekCogRichText and max size 1000 on a PDA with more memory.

The parameters are stored by the Cognima Server when the application isadded to the device. When new objects or updates for that class areplaced in the Cognima Server Change-Log for that device they areprocessed according to the smart parameters. This might involve, forexample, truncating text, converting Unicode text to narrow text orconverting image formats.

It is important for data integrity that the object held in the CognimaServer database be a copy of the object as it was generated. Even if yousee a cut down version on a device you can effectively manipulate thecomplete version on the Cognima Server.

5.1.7 Class Versions

We have the concept of a class version which is defined by a 4 byteunsigned integer. A new class version may add properties to the end ofthe old class, but it may not change or remove existing properties, orinsert new properties between existing properties. This should allowinteroperability between versions. Class definitions with differentsmart property parameters are not different versions.

5.2 Passing User Data Around

Cognima utilises the idea of class metadata to minimise the data thatneeds to be copied around between databases. Class metadata isessentially an array of property metadata. Property metadata is aproperty id, a property type, a smart type and a max size.

User data is transferred as a stream with no formatting informationother than a class id. This stream is parsed by looking up the classmetadata. So if a stream is received for class 6 and the class metadatafor class 6 says that property 0 is a KcogTypeUInt and property 1 is akcogTypeStr, then you know that the first 4 bytes of the stream shouldbe interpreted as an unsigned integer, the next 4 bytes should beinterpreted as an unsigned integer holding the number of characters n inthe succeeding string, the next n (times 2 if Unicode) bytes hold thecharacters in the string etc.

Client devices declare to the Cognima Server the classes that theysupport. This enables the device to subsequently send up only raw userdata (with a header containing class id, object id and a few otherthings) and hence minimises bandwidth requirements. This can becontrasted with, for example, XML reliant systems that are far morebandwidth hungry.

The client device class declarations also contain the smart propertyparameters so that the Cognima Server can sculpt the data for thedevice. It is worth emphasising that the meaning of a property is hardcoded into an application. The class metadata states that property 2 inclass 7 is a string with max length 30 characters. It is the code in theapplication that interprets property 2 in class 7 as the name of afootball team.

5.2.1 Data Replication Issues in More Depth

Data is held in Objects that are created on client devices and theserver these devices connect to (known as the Cognima Server). Theseobjects and any changes made to them are replicated between the clientdevices and the Cognima Server.

The design of the replication process allows:

A set of objects to be defined that will be replicated so that the sameset of objects will be held on a Cognima Server and all the clientdevices that are logged on to that server for a given user. New objectscreated on any device or the server will be replicated to all otherdevices. Changes in any property of an object will be replicated to alldevices.

Only the minimum data to be transmitted across the network for a givenupdate since only changes in data are sent from clients to the CognimaServer or vice versa.

A key part of the design was to not require times of modification to bekept for each property of an object on the client device as updatingthese on constrained client devices is slow and keeping a last modifiedtime for each property in an object would take a lot of space.

On the Cognima Server storing modification times for all properties ofan object is fine as the server has enough storage space and processingpower to deal with this.

5.2.2 Metadata

In order for the system to work it needs a clear idea of what propertiesare defined for a given class of objects. This is done by providing theprogrammer with a few C++ compiler macros that allow definition of theclass metadata.

The definition of the properties to be used in a class result in a ClassMetadata definition. This definition tells the CRE (Cognima recognitionengine) what type a given property is and allows it to pack and unpackan object or a property for transmission over a data link. In order forthe CRE system to work all clients and the server must have the sameclass metadata definition. Thus the following occurs:

When a new Metadata definition is declared on a client device it is sentto the Cognima Server and from there the Cognima Server will send it toall other clients.

When a new Metadata definition is declared on a Cognima Server thedefinition is sent to all client devices.

When a new client device logs on to a Cognima Server for the first timeall of the metadata definitions are sent to that device before anyobjects are sent.

In all of the above cases a future optimisation may be made so that theCognima Server only sends the metadata definition to clients who accessthe class (and the specific properties) the metadata refers to.

5.2.3 ChangeLog

The purpose of the ChangeLog is to record any changes that have occurredsince the client device last connected to the Cognima Server (or theCognima Server to the client device). Using Cognima APIs, applicationsconnect to the CRE and can cause objects to be created or deleted, or aproperty in an object to be changed. These changes are added to aChange-Log on the local device as they are made together with the timethe change was made. Objects are given unique identifiers when they arecreated so that a given object can always be identified.

In the same way, creation and deletion of objects and changes to objectproperties by applications running on the Cognima Server result in thechanges being added to all the Change-Logs of all the client devicesregistered to that user on the Cognima Server. The time of changes arerecorded for each object or property.

ChangeLogs can be built in two ways:

As the new objects are created and properties are changed (this isnormally the case for client devices)

Or they can be built on demand when they are needed by using the lastmodified times of objects and properties if these are stored on thesystem (in some circumstances, this method may be used on the CognimaServer instead of the above method).

5.2.4 Replication

When a client device has items in its ChangeLog to send it will connectto the Cognima Server (and likewise for the Cognima Server connecting tothe client device). By default, the items in the ChangeLog are sent inthe order in which they were added to the ChangeLog, however they may bere-prioritised immediately before sending to provide for premiumservices, urgent data and so on. Items transferred are:

A metadata definition including the type of each property of a givenclass of objects.

A new object that has been created—with the contents of the propertiesof that object.

A property has been changed—with the new value of the property.

An object has been deleted.

In all the above cases the appropriate IDs are sent to identify theobject, class and properties involved. All ChangeLog items are markedwith the time the item was added to the ChangeLog. These times arealways local machine times and are resolved into GMT by the TimeManagement approach described in Section 6.2.

When a client device receives ChangeLog items from a Cognima Server:

When a client device receives a new object message from a Cognima Serverit adds this new object to its local database.

When a client device receives an object deletion message from a CognimaServer it marks the object as deleted in its local database.

When a client device receives a property change it is always assumedthat the Cognima Server is authoritative on the current state of thedatabase and so the change is always made to the value of the propertyheld in the local database.

A Cognima Server receives ChangeLog items from a client device:

When a Cognima Server receives a new object from a client device it isadded to the Cognima Server database and also added to all theChange-Logs of the client devices registered to that user, apart fromthe Change-Log of the machine that sent the new object in the firstplace.

When a Cognima Server receives an object deletion from a client devicethe object is marked for deletion and an object deletion message isadded to all the Change-Logs of the devices registered to that userapart from the Change-Log of the machine that sent the object deletionin the first place.

When a Cognima Server receives a property change it compares the time ofthe change to the current time held for that property on the CognimaServer. If the time of the property change is later than that held onthe Cognima Server the property value is changed in the server databaseand this change is also added all the Change-Logs of the client devicesregistered to that user—including the one of the machine that sent inproperty change (in case another object update has been sent to thatmachine in the meantime). If the property change was not later than theone held on the Cognima Server no change is made as the stored propertyvalue is more recent—but the value is added to the list of old propertyvalues on the Cognima Server so that a user can retrieve it later ifrequired. When times are compared the Time Management approach describedin Section 6.2.below is used.

When a device first connects to a Cognima Server it will be sent allclass metadata definitions and then all the objects in the database forthat user. The Deletion messages generally just mark an Object fordeletion. Actual removal of the object from the database may occur lateron once all objects referring to that object have also been deleted.

5.2.5 Optimisations

An optimised version of the above replication protocol allows foraggregation of the entries in the ChangeLog. If a ChangeLog (in theCognima Server or on a client device) has not yet been replicated, and asubsequent entry is added, then existing entries can be scanned topotentially reduce the number of entries that need to be replicatedduring the next connection:

if the new entry is an update to a property that is already scheduledfor update then only the later entry need be retained

if the new entry is an object deletion then all property updates forthat object can be removed from the ChangeLog

if the new entry is an ‘undelete’ command and the original deletion isstill in the ChangeLog then the two entries can both be removed from theChangeLog

6. Core Algorithms

6.1 Handling Endian-ness

Operating systems are fundamentally little endian or big endian which isa choice of the byte order in which numbers and strings are stored. Iftwo computers which have different endian-ness have to communicate thenone of the computers will have to switch the endian-ness of its datapackets. In the Cognima environment the Cognima client software uses thesame endian-ness as the host client device. The Cognima Server has todetermine the endian-ness of the client device (it uses a referencevalue in the first packet of data from the client) and then convert thesubsequent incoming data if necessary to maintain consistent endian-nessin the Cognima Server. The Cognima Server also has to convert anyoutgoing data it sends back to the client device.

6.2 Synchronising System Times

Different devices will inevitably have slightly different system times.Changes that are sent from client devices to the Cognima Server arestamped with the device system time at the time of the change. It is upto the Cognima Server to resolve the times on different devices so thatit can judge the order in which changes took place and record thecorrect update.

The logon of a device contains the current device time. The CognimaServer should be able to compensate for the latency of the network andcompare the login time with its own system time. This will give it adelta between the device time and the Cognima Server time. This deltacan be applied to further times sent up by the device in that session.

The Cognima Server can compare deltas in successive sessions from adevice to determine clock ‘creep’ on the device or changes of time zone:it cannot be assumed that all the client devices in the system haveclocks that are well synchronised to each other:

Clock times drift on devices depending on the device's clock accuracy.

Some users like to set clocks 5 minutes early for example.

Some users will make changes to clocks to account for daylight savingrather than adjusting the locale settings (and some OSes may not providelocale features anyway forcing the user to change the clock directly).

To get round this problem, the server will be responsible for adjustingtimes used by the client device to GMT when comparisons are made on theServer, and from GMT to the equivalent time for the client device whenmessages are sent from the Cognima Server to the client device.

The client device will tag all the items in the ChangeLog with timesobtained from the local clock—as far as the client device is concernedit only ever deals in time based on the client device's own clock. Eachtime the client device connects to the Cognima Server it sends its viewof the current time as given by the clock on the client device. Fromthis the Server can work out:

What the delta to GM is

If there has been any drift in the mobile device clock since the lasttime it logged on since the server keeps a record of the last delta toGMT and when the last connection was made and therefore can comparethese. If there is drift the server can adjust all times sent by themobile device pro-rata.

For example the table below shows a pattern of events with a clientdevice connecting to a Cognima Server. The Client device's time is 5minutes slower that the Cognima Server and is loosing a minute everyhour (an extreme case to show the point). Also to show the point we willassume that from 09:00 to 12:00 the user is on a plane and out ofcontact with the Cognima Server so it does not connect during this time:Client Cognima Server Action Device Time time (GMT) Client deviceconnects to 09:00 09:05 Cognima Server A change is made to property A10:00 X A change is made to property B 11:00 Y Client device connects to12:00 12:08 Cognima Server

In order to work out if the property changes were made before or afterthe time stored on the Cognima Server the times X and Y need to beworked out. From the information above the Cognima Server knows thatwhen the client last connected it was around 3 hours ago and at thatpoint the time difference was 5 minutes whereas now it is 8 minutes.Thus, assuming the clock drift happens linearly, the Cognima Server canwork out that the device is 5 minutes behind GMT and that the clock isdrifting back a minute every hour.

From this is it possible to work out that the time the client deviceknows as 10:00 for the property A change needs to have 5 minutes addedto it for the initial drift, plus one minute for the extra drift thatoccurred in the hour till that property was changed.

Likewise Property B needs to be adjusted to 11:07—the 5 minutes initialdrift plus 2 minutes since two hours elapsed from 09:00 to 11:00 whenthe property was changed.

In practice the delta to the time between the client device time and GMTmay be minutes, but the drift will be in the order of fractions ofseconds per hour.

6.2.1 Time Adjustments

As well as the delta to GMT and any drift in the client device clock,users can also change the time on the client device. They may do this toreset the time to the correct local time (we can give the user theoption to have this happen automatically but some users may want to keeptheir own control of their client device time—e.g. they like to have theclock set 5 minutes fast). They may also make adjustments to reflect achange of local time (i.e. daylight savings or changing timezone). Thegoal is that the user can change the clock on the device to any timethat suits the user and the device simply takes account of this.

When the user makes a change to the client device time most operatingsystems will report this change (for systems that don't do this the timecan be polled say every minute to check for such a change). On detectinga change in time the client device will work out the delta between thenew time and the time as it was before the change. For example this maybe a change of plus one hour as a user moves timezone. The client devicestores this time difference as the Adjust Time which it saves for thenext connection to the Cognima Server. The client device also goesthrough every entry in the ChangeLog and updates all times in the log byAdjust Time. This ensures that the entries in the ChangeLog are alwaysrelative to the local time on the client device.

Several such adjustments could be made between connections to theCognima Server—each time the amount of the time change is summed withthe Adjust Time and the ChangeLog updated so that the times in the logare all relative to the local time on the client device.

When the client device next connects to the Cognima Server the clientdevice sends at logon the stored Adjust Time—i.e. the amount by whichthe client device clock has been adjusted backwards or forwards sincethe last connection. The Cognima Server can then remove this amount fromthe time from the delta to GMT and drift calculation.

6.2.2 GMT to Client Device

The same set of calculations can be made in reverse to convert the GMTtimes of changes made on the Cognima Server to the correct local timefor a given client device.

6.3 Adding an Application

An application will use one or more classes to hold user data. Thedefinition of the class is hard coded into the application. The versionof the class is co-ordinated by releases of the application.

Say that a statistics application uses a Footballer class to hold dataabout footballers. When the application starts on a client device forthe first time, it inquires from the device what version of theFootballer class the device already holds. If the version on the deviceis the same as the version that the application has been hard coded touse then nothing more need be done.

If the device holds a newer version of the Footballer class, then theapplication needs to be robust enough to cope with more properties thanit expected. (MTis situation would arise if you had a class being usedby multiple apps and for some reason you installed an older version ofone of the apps. This should be rare: ideally interdependent apps shouldbe upgraded together.)

If the device holds an older version of the Footballer class (or noversion at all) then the application's version of the Footballer classshould replace it. The new version is sent up to the Cognima Server. TheCognima Server therefore maintains a list of versions of classes used onall devices.

The web portal pages will be the equivalent of the hard-coded deviceapplication. The web can extract objects from the database according tothe latest version of the class, and if there are more properties thanit was hard coded to expect it can ignore them. Therefore the web doesnot need to declare class versions.

6.4 Change-Log Optimisation

The Cognima Server maintains Change-Logs for all devices listing changesthat will be sent to the devices when the devices next connect. Therewill be optimisations that can be made to the Change-Logs, for example:

If>2 updates to the same property are queued in the Change-Log then onlythe last need be kept.

If a deletion is queued for an object then any updates ahead in thequeue may be removed.

If an update is queued for an object then any delete ahead in the queueshould be removed.

If a device registers a new application there could potentially be verymany objects to send down to it (e.g. message history). The Change-Logshould only have a sensible number of objects added to it (e.g. the 20most recent messages).

7. Ghosting, Resurrection, Pinning and Withdrawal

The space available on a client device to hold user data will typicallybe orders of magnitude less than the space available on the server. Thedevice needs to hold a subset of data and the user should have to do aslittle work as possible to maintain this subset. Ghosting and withdrawalare tools to aid this.

A class definition may include flagging certain properties as‘ghostable’. This means that if the object is ghosted those propertieswill be nulled, freeing room on the client device. Ghosting is doneautomatically on the device. The decision about which objects to ghostis made by following a ‘ghosting rule’ and applying the rule whenever anobject is created or updated. The rule defines the maximum number of aselection of objects. When the maximum is exceeded the objects in theselection at the bottom of a sort order are ghosted.

For example, the class might be messages, the selection might bemessages in the inbox, the sort order might be by date/time and themaximum number might be 50. If there are 50 messages in the inbox and anew message arrives, the oldest message in the inbox is ghosted.Ghosting may remove the message body but leave enough header informationfor the message to be recognised.

Withdrawal (also known in the past as auto-deletion and removal) issimilar to ghosting but works by removing the entire object, not justpart of it.

Neither ghosting nor withdrawal are notified to the Cognima Server. Theyare purely local to the client device. Therefore different devices mayhave different numbers of objects. The data on the devices is stillfundamentally in sync, but the devices hold different data subsets.

If the user wants to resurrect a ghost then a request is passed from theclient to the Cognima Server for the object to be resurrected. A refreshof the object is sent down to the device and the object is put back tonormal.

Individual objects can be pinned. A pinned object is never ghosted orremoved. Pinning can be chosen by the user, or it can happenautomatically. For example, an object that is resurrected isautomatically pinned.

8. User Replication—Sharing Objects

There are many applications for which we envisage it will be useful forusers to be able to share objects. The general way that this will workis: A user needs to know the Cognima address of users that he may wantto share objects with. It is more appropriate to discuss the retrievalof these addresses in detail in the Cognima Server architecture. Here weassume that such a list is available.

A set of one or more Cognima addresses is attached to the object whichis to be shared. The object can be set to read-only (so the people youshare it with cannot modify it). When the Cognima Server receives thenew object (or receives an update to it) from the web or a client deviceit replicates it as normal.

It also looks up the list of ‘sharees’ Cognima addresses. It marks theobject with an originator id (i.e. the Cognima address of the objectowner+the object id) and sends it to the sharees. The sharee users mayexist on the same Cognima Server or be on different Cognima Servers. TheCognima Server of the sharee receives the object. If it is a new objectit assigns a new object id (keeping note of the originator id). If it isan update it finds the object using the originator id. If the sharee isallowed to update the object, the update can be replicated back to theobject owner using the originator id.

9. Displaying Data

Conventional small devices like PDA tend to have simple filing systemsthat allow applications to read and write data to some kind of storagethat will keep the data when the application is not running. Generallythese programs will tend to read in the available set of data and thenprovide a user interface to display the data on the screen. This hassome disadvantages:

Reading in the data when the program starts takes time

The application needs to store all or some of the data in memory meaningit is now occupying more memory on the client device

Allowing more than one application to access the same set of databecomes non-trivial

Similar code to read and manipulate the data appears in severalapplications that run on the device.

The Cognima approach is different:

Data is stored in an Object Database that can be accessed by severalapplications

A Cognima application does not read in all the data it deals with from adatabase. Instead it creates a selection—a subset of the data which itis currently interested in. In general this selection matches the datathat is currently being displayed on the devices screen. Thus only thedata currently being used by the application is held in memory—saving alot of memory space.

All of the work of storing, sorting and indexing the data is done by theObject Database and so this functionality does not need to be repeatedin each application.

When changes need to be made to data in an application, the applicationnever directly updates its own display of the data. Changes will updatethe properties in an object or create or delete an object. A change tothe data could be made by another application or an update received froma Cognima Server due to the data being changed on another machine.

When an application sets up a selection it gives a list of criteria bywhich data is either included or excluded from the selection—because ofthis the Cognima Replication Engine can tell which applications tonotify when a object is created, deleted or updated.

When an update needs to be sent to the application, code in theapplication linked to the selection that contains this data is calledand in this way the application can respond to the changes that havebeen made.

When selections are set up, the application can also specify how thedata is sorted and if only a small window on the sorted list of data isrequired (known as a view).

This approach is similar to the screen re-paint approach used to redrawgraphics screens on Windowing systems. When an area of the screen needsrepainting the application that is responsible for that bit of screen iscalled to repaint the screen.

9.1 Example

A client device may have a contacts application running on it—thisdevice replicates data with a Cognima Server connected to other clientdevices also running contacts applications. A class of object is definedfor a Contact that contains names and phone numbers and these arereplicated to all the devices of a given user.

An application on one device may have a display that shows all contactsby beginning letter—for example the interface allows the user to press aD button to show all the names beginning with D. This application willset up a selection that contains objects:

Where the class is defined as Contacts

Where the name begins with the selected letter (e.g. D)

When the selection is defined the application also defines code to becalled by the CRE when objects are added, deleted or updated. When theselection is first set up this code will be called back with the firstset of objects that fulfill the above criteria.

If the application was asked to create a new contact with a namebeginning with D the application would create the object but do nothingelse. The CRE would detect the new object and call back the selectioncode to notify it of the new object.

Likewise is a new Contact object was created on another device and wasreplicated to the client device—if the name of that Contact began with Dthe application would be notified.

9.2 Sorting

Data in selections generally needs to be sorted—often so that whendisplayed users can see data in a logical format. When a selection isdefined the sorting order can be specified: the properties to sort on,in what order and what sorting algorithms to use.

9.3 Views

There may be many items of data in a selection. Commonly when the datais being displayed it may not all fit on the screen and so the user willneed to scroll up and down the data. A view provides this functionalityby specifying the number of items of data the selection wants to dealwith and the number of the first item of data out of the complete listof data the application wants to appear in the selection.

Views are important because they allow an application to limit theamount of data it stores locally to be limited to just the amount neededto display on the screen this reducing unnecessary duplication of data.

9.4 Efficiency

Cognima has made some efficiency optimisations in how the data istransferred between the Cognima server and client application—whenmultiple data changes are made the data is sent in blocks and then theapplication informed that the changes are complete so that theapplication only needs to update its user interface once.

9.5 Example

As an example we will define a selection called ContactSelection. Thisis the code that the framework will call back whenever a change is madeto any of the selected objects. In the Cognima framework this isimplemented as an object which you derive from the COdbSelect templatedclass—specifiying the type of object you want to have in the selectionas the template argument. class CContactSelect: publicCOdbSelect<CContact> { public:  CContactSelect(COdb *aOdb);  voidObjectAdded(CContact *aObject);  void ObjectUpdated(CContact *aObject); void ObjectRemoved(const TOdbObjectId aObjectId); private:  boolListContacts( ); };

The methods ObjectAdded( ), ObjectUpdated( ) and ObjectRemoved( ) arecalled by the framework whenever respectively an object is added,updated or removed. When you implement the Selection class you don'tneed to implement all these methods if you do not want to take instanceaction on any of these events—in some cases you may set up a selectionto keep a list of a certain set of objects but only check that list onsome other event and so the above methods would not be required.

We have defined one extra private method called ListContacts( )—thiswill list all the current contacts held by the selection.

Here is the implementation of this class:

CContactSelect::CContactSelect(COdb *aOdb)

: COdbSelect<CContact>(aOdb)

{

}

void CContactSelect::ObjectAdded(CTestContact*aContact) { OdbLog(OdbLogApp,L“New contact added: ” << aContact->  GetName( )); ListContacts( ); }

void CContactSelect::ObjectUpdated(CTestContact*aContact) { OdbLog(OdbLogApp,L“Contact updated: ” << aContact->GetName( )); ListContacts( ); }

void CContactSelect::ObjectRemoved(const TOdbObjectId aObjectId) { OdbLog(OdbLogApp,L“Contact deleted - Id: ” << aObjectId); ListContacts( ); }

void CContactSelect::ListContacts( ) {  OdbLog(OdbLogApp,L“Contactslist:”);  for (unsigned long Index=0; Index<iResult.size( ); Index++)  {  CTestContact*Contact=iResult[Index];  OdbLog(OdbLogApp,Contact->GetName( ) << “, ”    << Contact->GetPhone() << “, ”    << Contact->GetEmail( ));  } }

The constructor simply calls the default COdbSelect constructor. TheObjectAdded( ), Updated( ) and Removed( ) methods print out what changewas made and then call ListContacts( ) to show what the current contentsof the list is.

The ListContacts( ) shows how the current list of object held by theselection can be accessed. The current list of pointers to objects isheld in a container class called iResult—this can then be accessed bynormal container class integrators. In this case we simply go throughthe list and print all the objects in the list.

1. A method of replicating data between computing devices which each uselocal clocks, in which a first computing device is given responsibilityfor determining whether data that is time stamped by a second computingdevice, using a clock running on the second device, is replicated ornot; wherein that first device can compare the time stamp from thesecond device against a time signal from its own local clock todetermine a delta and use that delta to deduce the correct delta toapply to time stamps associated with later data from the secondcomputing device.
 2. The method of claim 1 in which the local clock ofthe first computing device is set to an absolute, externally derivedtime.
 3. The method of claim 1 in which the delta deduced by the firstdevice assumes that the local clock in the second device is drifting ina linear manner.
 4. The method of claim 1 in which, if a deliberate timeadjustment to accommodate a change of time frame is made to the localclock on the second device, then the second device stores a record ofthis time adjustment and provides it to the first computing device. 5.The method of claim 4 in which the second device alters one or more timestamps stored on it by the amount of the deliberate time adjustment sothat the time stamps remain correct in the new time frame.
 6. The methodof claim 1 in which the first computing device uses its knowledge of thecorrect deltas relative to its own clock in order to apply correctly atime dependent arbitration rule to determine which of severalinconsistent data replication requests it should action.
 7. The methodof claim 1 in which the time stamp applied by the second device is notpart of a database record containing the data to be replicated.
 8. Themethod of claim 7 in which the time stamp resides in a change log whichincludes only the minimum data needed to identify a change to data.
 9. Acomputing device capable of replicating data and having responsibilityfor determining whether data that is time stamped by a second computingdevice, using a clock running on the second device, is replicated ornot; wherein the device can compare the time stamp from the seconddevice against time signals from its own local clock to determine adelta and use that delta to deduce the correct delta to apply to timestamps associated with later data from the second computing device. 10.The device of claim 7 being a server and the second computing devicesacting as clients of that server.